Dual-space disentangled-multimodal network (DDM-net) for glioma diagnosis and prognosis with incomplete pathology and genomic data.

Objective. Effective fusion of histology slides and molecular profiles from genomic data has shown great potential in the diagnosis and prognosis of gliomas. However, it remains challenging to explicitly utilize the consistent-complementary information among different modalities and create comprehensive representations of patients. Additionally, existing researches mainly focus on complete multi-modality data and usually fail to construct robust models for incomplete samples.Approach. In this paper, we propose adual-space disentangled-multimodal network (DDM-net)for glioma diagnosis and prognosis. DDM-net disentangles the latent features generated by two separate variational autoencoders (VAEs) into common and specific components through a dual-space disentangled approach, facilitating the construction of comprehensive representations of patients. More importantly, DDM-net imputes the unavailable modality in the latent feature space, making it robust to incomplete samples.Main results. We evaluated our approach on the TCGA-GBMLGG dataset for glioma grading and survival analysis tasks. Experimental results demonstrate that the proposed method achieves superior performance compared to state-of-the-art methods, with a competitive AUC of 0.952 and a C-index of 0.768.Significance. The proposed model may help the clinical understanding of gliomas and can serve as an effective fusion model with multimodal data. Additionally, it is capable of handling incomplete samples, making it less constrained by clinical limitations.

Automatic classification of heart failure based on Cine-CMR images.

PURPOSE: Heart failure (HF) is a serious and complex syndrome with a high mortality rate. In clinical diagnosis, the correct classification of HF is helpful. In our previous work, we proposed a self-supervised learning framework of HF classification (SSLHF) on cine cardiac magnetic resonance images (Cine-CMR). However, this method lacks the integration of three dimensions of spatial information and temporal information. Thus, this study aims at proposing an automatic 4D HF classification algorithm. METHODS: To construct a 4D classification model, we proposed an extensional framework called 4D-SSLHF. It mainly consists of self-supervised image restoration and HF classification. The image restoration proxy task utilizes three image transformation methods to enhance the exploration of spatial and temporal information in the Cine-CMR. In the classification task, we proposed a Siamese Conv-LSTM network by combining the Siamese network and bi-directional Conv-LSTM to integrate the features of the four dimensions simultaneously. RESULTS: Experimental results on 184 patients from Shanghai Chest Hospital achieved an AUC of 0.8794 and an ACC of 0.8402 in the five-fold cross-validation. Compared with our previous work, the improvements in AUC and ACC were 2.89 % and 1.94 %, respectively. CONCLUSIONS: In this study, we proposed a novel self-supervised learning framework named 4D-SSLHF for HF classification based on Cine-CMR. The proposed 4D-SSLHF can mine 3D spatial information and temporal information in Cine-CMR images well and accurately classify different categories of HF. The good classification results show our method's potential to assist physicians in choosing personalized treatment.

Hierarchical multimodal fusion framework based on noisy label learning and attention mechanism for cancer classification with pathology and genomic features.

Classification of subtype and grade is imperative in the clinical diagnosis and prognosis of cancer. Many deep learning-based studies related to cancer classification are based on pathology and genomics. However, most of them are late fusion-based and require full supervision in pathology image analysis. To address these problems, we present an integrated framework for cancer classification with pathology and genomics data. This framework consists of two major parts, a weakly supervised model for extracting patch features from whole slide images (WSIs), and a hierarchical multimodal fusion model. The weakly supervised model can make full use of WSI labels, and mitigate the effects of label noises by the self-training strategy. The generic multimodal fusion model is capable of capturing deep interaction information through multi-level attention mechanisms and controlling the expressiveness of each modal representation. We validate our approach on glioma and lung cancer datasets from The Cancer Genome Atlas (TCGA). The results demonstrate that the proposed method achieves superior performance compared to state-of-the-art methods, with the competitive AUC of 0.872 and 0.977 on these two datasets respectively. This paper establishes insight on how to build deep networks on multimodal biomedical data and proposes a more general framework for pathology image analysis without pixel-level annotation.

PUNDIT: Pulmonary nodule detection with image category transformation.

BACKGROUND: Convolutional neural networks (CNNs) have achieved great success in pulmonary nodules detection, which plays an important role in lung cancer screening. PURPOSE: In this paper, we proposed a novel strategy for pulmonary nodule detection by learning it from a harder task, which was to transform nodule images into normal images. We named this strategy as pulmonary nodule detection with image category transformation (PUNDIT). METHODS: There were two steps for nodules detection, nodule candidate detection and false positive (FP) reduction. In nodule candidate detection step, a segmentation-based framework was built for detection. We designed an image category transformation (ICT) task to translate nodule images into pixel-to-pixel normal images and share the information of detection and transformation tasks by multitask learning. As for references of transformation tasks, we proposed background consistency losses into standard cycle-consistent adversarial networks, which can solve the problem of background uncontrolled changing. A three-dimensional network was used in FP reduction step. RESULTS: PUNDIT was evaluated in two datasets, cancer screening dataset (CSD) with 1186 nodules for cross-validation and (CTD) with 3668 nodules for external test. Results were mainly evaluated by competition performance metric (CPM), the average sensitivity at seven predefined FP rates. The CPM was improved from 0.906 to 0.931 in CSD, and from 0.835 to 0.848 in CTD. CONCLUSIONS: Experimental results showed that PUNDIT can improve the performance of pulmonary nodules detection effectively.

A Self-supervised Learning Based Framework for Automatic Heart Failure Classification on Cine Cardiac Magnetic Resonance Image.

Heart failure (HF) is a serious syndrome, with high rates of mortality. Accurate classification of HF according to the left ventricular ejection faction (EF) plays an important role in the clinical treatment. Compared to echocardiography, cine cardiac magnetic resonance images (Cine-CMR) can estimate more accurate EF, whereas rare studies focus on the application of Cine-CMR. In this paper, a self-supervised learning framework for HF classification called SSLHF was proposed to automatically classify the HF patients into HF patients with preserved EF and HF patients with reduced EF based on Cine-CMR. In order to enable the classification network better learn the spatial and temporal information contained in the Cine-CMR, the SSLHF consists of two stages: self-supervised image restoration and HF classification. In the first stage, an image restoration proxy task was designed to help a U-Net like network mine the HF information in the spatial and temporal dimensions. In the second stage, a HF classification network whose weights were initialized by the encoder part of the U-Net like network was trained to complete the HF classification. Benefitting from the proxy task, the SSLHF achieved an AUC of 0.8505 and an ACC of 0.8208 in the 5-fold cross-validation.

A Classification-Guided Segmentation Algorithm Based on Deep Learning for Epithelium Segmentation in Histopathological Images of Radicular Cysts.

In histopathological analysis of radicular cysts (RCs), lesions in epithelium can provide pathologists with rich information on pathologic degree, which is helpful to determine the type of periapical lesions and make precise treatment planning. Automatic segmentation and localization of epithelium from whole slide images (WSIs) can assist pathologists to complete pathological diagnosis more quickly. However, the class imbalance problem caused by the small proportion of fragmented epithelium in RCs imposes challenge on the typical automatic one-stage segmentation method. In this paper, we proposed a classification-guided segmentation algorithm (CGSA) for accurate segmentation. Our method was a two-stage model, including a classification network for region of interest (ROI) location and a segmentation network guided by classification. The classification stage eliminated most irrelevant areas and alleviated the class imbalance problem faced by the segmentation model. The results of 5-fold cross validation demonstrated that CGSA outperformed the one-stage segmentation method which was lacking in prior epithelium localization information. The epithelium segmentation achieved an overall Dice's coefficient of 0.722, and intersection over union (IoU) of 0.593, which improved by 5.5% and 5.9% respectively compared with the one-stage segmentation method using UNet.Clinical Relevance- This work presents a framework for automatic epithelium segmentation in histopathological images of RCs. It can be applied to make up for the shortcomings of manual annotation which is labor-intensive, time-consuming and objective.

Lung cancer subtype classification using histopathological images based on weakly supervised multi-instance learning.

Objective.Subtype classification plays a guiding role in the clinical diagnosis and treatment of non-small-cell lung cancer (NSCLC). However, due to the gigapixel of whole slide images (WSIs) and the absence of definitive morphological features, most automatic subtype classification methods for NSCLC require manually delineating the regions of interest (ROIs) on WSIs.Approach.In this paper, a weakly supervised framework is proposed for accurate subtype classification while freeing pathologists from pixel-level annotation. With respect to the characteristics of histopathological images, we design a two-stage structure with ROI localization and subtype classification. We first develop a method called multi-resolution expectation-maximization convolutional neural network (MR-EM-CNN) to locate ROIs for subsequent subtype classification. The EM algorithm is introduced to select the discriminative image patches for training a patch-wise network, with only WSI-wise labels available. A multi-resolution mechanism is designed for fine localization, similar to the coarse-to-fine process of manual pathological analysis. In the second stage, we build a novel hierarchical attention multi-scale network (HMS) for subtype classification. HMS can capture multi-scale features flexibly driven by the attention module and implement hierarchical features interaction.Results.Experimental results on the 1002-patient Cancer Genome Atlas dataset achieved an AUC of 0.9602 in the ROI localization and an AUC of 0.9671 for subtype classification.Significance.The proposed method shows superiority compared with other algorithms in the subtype classification of NSCLC. The proposed framework can also be extended to other classification tasks with WSIs.

A Weak Supervision-based Framework for Automatic Lung Cancer Classification on Whole Slide Image.

Classification of normal lung tissue, lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC) by pathological images is significant for clinical diagnosis and treatment. Due to the large scale of pathological images and the absence of definitive morphological features between LUAD and LUSC, it is time-consuming, laborious and challenging for pathologists to analyze the microscopic histopathology slides by visual observation. In this paper, a pixel-level annotation-free framework was proposed to classify normal tissue, LUAD and LUSC slides. This framework can be divided into two stages: tumor classification and localization, and subtype classification. In the first stage, EM-CNN was utilized to distinguish tumor slides from normal tissue slides and locate the discriminative regions for subsequent analysis with only image-level labels provided. In the second stage, a multi-scale network was proposed to improve the accuracy of subtype classification. This method achieved an AUC of 0.9978 for tumor classification and an AUC of 0.9684 for subtype classification, showing its superiority in lung pathological image classification compared with other methods.

Adrenal Tumor Vessels Segmentation Using Convolutional Neural Network in Computed Tomography Angiography.

The adrenal glands are important endocrine glands in humans. They are in complex environments with thin vessels around them. It's meaningful to get the accurate dissection before surgery. However, images used in hospitals are now unable to help doctors with many surgeries, which are produced by digital subtraction angiography. In this study, we used a 3D U-Net model to segment the adrenal tumor vessels in 3D computed tomography angiography slices. The model was evaluated by dice similarity coefficient (DSC) and mean intersection over union (MIoU) with the manually labeled ground truth. The DSC in this model is 94.69% and the MIoU is 90.22%.